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A Time/Motion Compensation Engine for HMDs 

The foilowing discussion describes the Time/Motion Compensation Engine. This 
is a means of providing a user witti a stable frame of visual refereiice for 
aspects of a visual telepresence loop which would otherwise induce nausea 

This information is proprietary to Xiphos Technologies Inc. and may not be 
disclosed without prior permission. 
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1. Introduction 

A typical application using the Time/Motion Compensation Engine (T/MCE) is 
depicted in Figure 1 beloyj/. It consists of an operator using a head tracker to 
remotely control a camera as a slave. The video returned from tha camera is 
presented to the operator in a Head Mounted Display (HMO). If the control and 
feedback paths are delayed (eg. 0.1 sec to 5 sec), presentation ot the video 
directly in the HMD would induce nausea in the operator, due to the lack of co- 
ordination between head motion and visual cues. 
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Figure 1, Delav-Comoensated HMD Video System 
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Ac wCp:ctQCf Figure 1, 2 corrp'Jter ha? be^r? Ins'^rt?^ in ♦h^ f^th '^th^ video 
feedback from the camera. This computer can move the video image relative to 
the operator's HMO field of view. This is done in either software or hardware, 
and can be done at the same rates as the operator's head motion. The following 
discussion will descrit>e how this ability can be used to provide the operator 
a stable frame of visual reference for three aspects of telepresence which are 
capable of inducing nausea: 



1 . He ad motion without c o r r esponding camera m ot i o n. 

2. Camera motion without corresponding head motion, arnl 

3. Communication delays between head motion and the resultant corre5p>ondlng 
camera motion. 
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2. Field of View 

In the Figure 2, we see the camera's view of the Earth. The field of view (FOV) 
of the HhAD is depicted by the dashed lines. This is the area seen by the 
operator in his display. The gray area around this depicts the range over which 
the computer can re-locate the video image. This gray area is« in fact, 3* 
dimensional and descnbes the interior of a sphere. For this discussion, we will 
assume it is flat and head motions are small. 




Figure 2, Initial Camera View 
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Thus, the computer can move the Image so that it is not co-located with the part 
of the region displayed on the HMO. If the image is removed from Ihe HMD's 
field of view, it is replaced by a neutral gray tone. This is depicted in Figure 3 
below, where the video image has been moved up and to the right. 




3. Time/Motion Compensation 

The method of compensating for time and motion incongruities in the HMD will 
be presented in three parts: 

1. Head motion without corresponding camera motion, 

2. Camera motion without corresponding head motion, and 

3. Communication delays between head motion and the resuftacit corresponding 
camera motion. 

The third of these Is actually a result of the first two, and does not require added 
functionality. 
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Compensation for Head Motion 
To provide a stable frame of visual reference for the operator, it is first 
necessary to permit free head movement, without depending on the remote 
camera to provide an instantaneous response. The worst case for this is having 
the camera faed, and the operator moving his head. In this scenario, the image 
must remain fixed in one location of 3-D space, as perceived by ths operator. 

mat is, in lumifiQ hte hedd, he would I60k awdy ffom the imaee ds if looking — ^ 

away from' a window frame in front of him. 

The stable frame of reference is provided by having the contents of the window 
remain fixed in 3-0 space. 

The image already shown in Figure 3 is the one that would result from the 
Q following example case. Starting from Figure 2, the camera is locked in a fixed 

1^ direction and the operator moves his head down and to the left. The computer 

I f| reading the head tracker would stkte the image up and to the right in real time. 

This provides the operator with a sensation that he is looking at tl^^e Earth 
lV through a v^ndow in front of him. He has looked down and to the left, thus the 

iy scene in the window is the same, but the window frame is now located at>ove 

ry and to his right Because the frame and its contents have moved appropriately 

s for his head motion, there is no nausea. 

C3 

The displacement of the window Is continuous and smooth; the rt^ulred speed 
f U of displadrig the window frame is the same as that of a generating a virtual 

f U environment for a VR HMO user. However, no graphics are being generated. 

\a Since the only operation of the computer is moving a wdeo window relative to its 

\3 display centre, this can be done very fast in hardware or software. Processing 

speeds are thus reasonable. 

If large head motions (or a r)arrow-angle video image are antidpated). it may be 
desirable to add horizontal and vertical pinstripe reference lines to the gray area 
to help the operator maintain orientation if he looks so far from t/ie window fr^ne 
that no part of the image is visible. 
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3 J. Compensation for Camera Motion 

To provide a stable frame of visual reference for the operator* it is also 
necessary to permit free camera movement, without having input thie 
corresponding head-mcMion command first In Itself, this capability is applicable 
to mobile camera platforms, where the platform may be subject to unanticipated 
motion. In cases where the camera is stabilized on a gimbal or where the 
operator is on a motion base, this capability can provide a very high frequency 
response which would otherwise incur cost, power and weight penalties In a 
mechanical system. This high frequency response may be complementary to 
the mechanical response of the gimbal or motion base. 

The baseline scenario is one where the operator's head is fixed, and the camera 
is moving independently, in this scenario, the contents of the scaie must 
remain fixed in one location of 3-D space, as perceived by the operator The 
window frame, however, must move with the camera. That is, as the camera 
^ moves, the operator imII perceive the field of view moving alMwt, as if the window 

frame in front of him were moving. The elements of images seen through the 
m frame (such as the Earth) would remain fixed in 3-D space. 

4U 77)0 stable frame of reference is provkied, once again, by having the contend of 

ru Me window remain Ibced in 3-D space, as perceived by tiie operator. 
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^ The next example descrities the algorithm's ability to move the mndow frame 

^ according to the camera's pointing angle. Here, the operator's head is fixed, and 

he is looking to the left and below the Earth through the remote camera. This is 
shown in Figure 4 below. 
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Now. (for whatever reason) the camera pans up and to the right vm^out tiie 
operator moving his head. The \ndeo data t>ejng downlinked is accomp^i^ 
data describing the instantaneous pointing direction of the gimt>al mount, and 
the motion of the platform. The computer reads this data, and moves the image 
(and window frame) up and to the right according to where the camera is 
pointing. The resulting presentation to the user is that of Figure 5« 
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Fi«^tire4. Init ial vi«^wBfelow and to the LeQ 




Aaain the user does noi experience nausea, this time because wndow 
^me S^ved to foto?^ video image. Parts of the scene that were v,s.b e 
to Se S^trbeSi the camera moved are -^^^'^^^'-^XTZ^^:^^ ^'^ 
field of vie«;. The image of the Earth and Space does not "slide .n the HMD 
frame; it just l)eoomes partly grayed out 
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3«d. Compensation for Communications Delay 

The net effect of a communications delay in an HMD master-$lave arrangement 
is the combination of the two previous effects. When the operator makes a head 
motion, there is initially no response from the camera. This is momentarily 
equivalent to the camera-fixed case. After a round-trip time delay, the images 
resulting from the camera's motion will be seen. This is equivalem to 
independent camera motiQn. ! 



Compensation for time delay is thus acheived by continuously oombinrng the 
window frame displacements mandated by head motion, with those indicated by 
the gimbal motion sensors. 

Consider the final example. Figure 6 is the initial view of the Eartli with 
fj^ operators head and camera at rest The operator turns his head lo the left and 

P stops. If there were no delays from transmission or camera motion, the 

iA instantaneous (and desired) outcome would be that of Figure 7. 
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With a delayed video response, the following happens: the operator's motion to 
the left causes the window frame and image contents to move to the right 
accordingly. The operator has the sensation of looking away from the window 
frame. As the camera returns its delayed response, the operator has the 
sensation of the window frame moving, and the scene beyond the window 
staying still in space; 
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IniCal Reld of View 




Head Turned to the Left 




Head Stopped. Camera Responding Camera Caught up to Head Position 
(Delayed) 

Figure 7, Seouencc for Time Delay Compensation 
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4. <;iimmary 

One limilalion of the concept must be kept in mind. Events seen by the camera 
will always be delayed by the downlink time before being displayed in the HMD. 
This method does not purport to convey images faster than the speed of light, 
and does not use any form of image synthesis or predictive logic to generate all 
or part of an image which !s being viewed by the camera, but has not yet been 
received by the operator. 



A good example would be viewing a spin^tabilized satellite in geos^i^nchronous 
ofbit, from a camera on a nearby free-flying servicer. The rotational position of 
the spinning satellite viewed by the operator would lag the actual position by the 
downlink time. If the operator were controlling a manipulator arm on the 
servicer, he would see the arm move in his HMD some time after issuing the 
command. This time wouM t>e the combiried uplink and downlink delays. 

^1 The following key points surrwnarize the concept 

1 • The view depicted witNn the window frame only moves relative to the HMO 
FOV when the operator's head moves. This is the key to providing a stable 
y frame of reference. 

ru 

9 2. The vnndaw frame moves relative to the HMD FOV in response to the camera 

□ motion, to locate the frame in 3-D space relative to the operator's head position. 

fU 3. The window frame also moves relative to the HMD FOV in response to the 

ry operator's head motion, to locate the frame and view in 3-0 space relative to the 

\5 operator's head position. 

4. All camera and head motions are co/itinaoiys/y transformed in 3-D space to 
provide a continuous, fluid response. 

5. The method is particularly appropriate for wide fields of view (in Hie camera 
and HMD), as ft will allow the operator to concentrate on the central field of view, 
while the gray window frame will only obscure penpheral areas temporarily. 

6. TTie method is suitable for translations and rotations. Provided the camera is 
gimbailed similarly to a human head, ttiere are minimal depth-of fiekj or parallax 
considerations. Wide angle distortions are transient 

7. Camera orientation can be sensed by gimbal position (on a fixed platform), or 
gimbal plus inertial orientation on a moving platform. Additionally, software 
image locking could further stablize the scene, particularly in cases where the 
camera is subject to motion vibration which cannot be damped by the gimbal. 
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